Fallout: Distributed systems testing as a service

All modern distributed systems list performance and scalability as their core strengths. Given that optimal requires carefully selecting configuration options, typical cluster sizes can range anywhere from 2 to 300 nodes, it is rare for any two clusters be exactly the same. Validating behavior of in this large space challenging without automation stretches across software stack. In paper we present Fallout, an open-source testing service automatically provisions configures clients, supports running a variety workloads benchmarks, generates reports based on collected metrics visual analysis. We have been Fallout internally at DataStax over 5 years recently open sourced support our work with Apache Cassandra, Pulsar, other source projects. describe architecture along evolution its design lessons learned operating dynamic environment where teams different products favor benchmarking tools.